Genetic Programming for Classification with Unbalanced Data

In classification, machine learning algorithms can suffer a performance bias when data sets are unbalanced. Binary data sets are unbalanced when one class is represented by only a small number of training examples (called the minority class), while the other class makes up the rest (majority class). In this scenario, the induced classifiers typically have high accuracy on the majority class but poor accuracy on the minority class. As the minority class typically represents the main class-of-interest in many real-world problems, accurately classifying examples from this class can be at least as important as, and in some cases more important than, accurately classifying examples from the majority class. Genetic Programming (GP) is a promising machine learning technique based on the principles of Darwinian evolution to automatically evolve computer programs to solve problems. While GP has shown much success in evolving reliable and accurate classifiers for typical classification tasks with balanced data, GP, like many other learning algorithms, can evolve biased classifiers when data is unbalanced. This is because traditional training criteria such as the overall success rate in the fitness function in GP, can be influenced by the larger number of examples from the majority class. This thesis proposes a GP approach to classification with unbalanced data. The goal is to develop new internal cost-adjustment techniques in GP to improve classification performances on both the minority class and the majority class. By focusing on internal cost-adjustment within GP rather than the traditional databalancing techniques, the unbalanced data can be used directly or “as is” in the learning process. This removes any dependence on a sampling algorithm to first artificially re-balance the input data prior to the learning process. This thesis shows that by developing a number of new methods in GP, genetic program classifiers with good classification ability on the minority and the majority classes can be evolved. This thesis evaluates these methods on a range of binary benchmark classification tasks with unbalanced data. This thesis demonstrates that unlike tasks with multiple balanced classes where some dynamic (non-static) classification strategies perform significantly better than the simple static classification strategy, either a static or dynamic strategy shows no significant difference in the performance of evolved GP classifiers on these binary tasks. For this reason, the rest of the thesis uses this static classification strategy. This thesis proposes several new fitness functions in GP to perform cost adjustment between the minority and the majority classes, allowing the unbalanced data sets to be used directly in the learning process without sampling. Using the Area under the Receiver Operating Characteristics (ROC) curve (also known as the AUC) to measure how well a classifier performs on the minority and majority classes, these new fitness functions find genetic program classifiers with high AUC on the tasks on both classes, and with fast GP training times. These GP methods outperform two popular learning algorithms, namely, Naive Bayes and Support Vector Machines on the tasks, particularly when the level of class imbalance is large, where both algorithms show biased classification performances. This thesis also proposes a multi-objective GP (MOGP) approach which treats the accuracies of the minority and majority classes separately in the learning process. The MOGP approach evolves a good set of trade-off solutions (a Pareto front) in a single run that perform as well as, and in some cases better than, multiple runs of canonical single-objective GP (SGP). In SGP, individual genetic program solutions capture the performance trade-off between the two objectives (minority and majority class accuracy) using an ROC curve; whereas in MOGP, this requirement is delegated to multiple genetic program solutions along the Pareto front. This thesis also shows how multiple Pareto front classifiers can be combined into an ensemble where individual members vote on the class label. Two ensemble diversity measures are developed in the fitness functions which treat the diversity on both the minority and the majority classes as equally important; otherwise, these measures risk being biased toward the majority class. The evolved ensembles outperform their individual members on the tasks due to good cooperation between members. This thesis further improves the ensemble performances by developing a GP approach to ensemble selection, to quickly find small groups of individuals that cooperate very well together in the ensemble. The pruned ensembles use much fewer individuals to achieve performances that are as good as larger (unpruned) ensembles, particularly on tasks with high levels of class imbalance, thereby reducing the total time to evaluate the ensemble. Publications Produced The following fully-referred papers were published during this Ph.D. 1. Urvesh Bhowan, Mark Johnston, Mengjie Zhang, Xin Yao. “Evolving Diverse Ensembles using Genetic Programming for Classification with Unbalanced Data”. IEEE Transactions on Evolutionary Computation (Accepted April 2012). 2. Urvesh Bhowan, Mark Johnston, Mengjie Zhang. “Developing New Fitness Functions in Genetic Programming for Classification with Unbalanced Data”. IEEE Transactions on Systems, Man, and Cybernetics (Part B), volume 42, issue 2. 2011. pp 406–421. 3. Urvesh Bhowan, Mengjie Zhang and Mark Johnston. “Ensemble Learning and Pruning in Multi-Objective Genetic Programming for Classification with Unbalanced Data”. Proceedings of the 24th Australasian Joint Conference on Artificial Intelligence (AI 2011). Lecture Notes in Artificial Intelligence. Vol. 7106. Springer. Perth, Australia, December, 2011. pp. 192–202. 4. Urvesh Bhowan, Mengjie Zhang, Mark Johnston. “Evolving Ensembles in Multi-objective Genetic Programming for Classification with Unbalanced Data”. Proceeding of the Genetic and Evolutionary Computation Conference (GECCO 2011). ACM Press. Dublin, Ireland. 2011. pp. 1331–1338. 5. Urvesh Bhowan, Mengjie Zhang, Mark Johnston. “A Comparison of Classification Strategies in Genetic Programming with Unbalanced Data”. Proceedings of the 23rd Australasian Joint Conference on Artificial Intelligence. AI 2010: Advances in Artificial Intelligence. Lecture Notes in Artificial Intelligence. Vol. 6464. Springer. Adelaide, Australia, 2010. pp. 243–252. (Nominated for the Best Student Paper award) 6. Urvesh Bhowan, Mengjie Zhang, Mark Johnston. “AUC Analysis of the Pareto-Front using Multi-objective GP for Classification with Unbalanced